4.5 Linear Models

#HypothesisTesting #Chi2Distribution #tDistribution #tTest #FDistribution #ZTest #LinearRegression #ANOVA

1 "Gaussian-Adjacent" Distributions

1.1 Chi Square Distribution

If $Z_{1}, \dots, Z_{d} \overset{i . i . d}{\sim} N (0, 1)$ , then $\begin{aligned} V & = \sum_{i = 1}^{d} Z_{i}^{2} \sim χ_{d}^{2} = Gamma (\frac{d}{2}, 2), \\ E V & = d, Var (V) = 2 d . \end{aligned}$
By CLT, $\frac{V - d}{\sqrt{2 d}} \Rightarrow N (0, 1) .$ So informally, $\frac{V}{d} \approx N (1, \frac{2}{\sqrt{d}}) \to 1.$

1.2 t Distribution

If $Z \sim N (0, σ^{2})$ and $V \sim σ^{2} χ_{d}^{2}$ , $Z ⊥ ⊥ V$ , then $\frac{Z}{\sqrt{\frac{V}{d}}} \sim t_{d} \Rightarrow N (0, 1) .$

1.3 F Distribution

If $V_{1} \sim σ^{2} χ_{d_{1}}^{2}$ and $V_{2} \sim σ^{2} χ_{d_{2}}^{2}$ , $V_{1} ⊥ ⊥ V_{2}$ , then $\frac{V_{1} / d_{1}}{V_{2} / d_{2}} \sim F_{d_{1}, d_{2}} \Rightarrow \frac{1}{d_{1}} χ_{d_{1}}^{2} as d_{2} \to \infty .$

If $T \sim t_{d}$ , then $T^{2} \sim F_{1, d}$ .

Recall $Z \sim N_{d} (μ, Σ), A \in R^{k \times d}, b \in R^{k}$ , then $A Z + b \sim N_{k} (A μ + b, A Σ A^{T})$ .

2 One-Sample T-test

$X_{i} \overset{i . i . d}{\sim} N (μ, σ^{2}) ⟺ X \sim N_{n} (μ \cdot 1_{n}, σ^{2} I_{n})$ . Want to test $H_{0} : μ = 0 vs H_{1} : μ \neq 0.$ Recall from this example the UMPU test is to reject for extreme $R = \frac{\sqrt{n} \overset{―}{X}}{| | X | |} = Corr (X, 1_{n})$ . And $T = \frac{\sqrt{n} \overset{―}{X}}{\sqrt{S^{2}}} = \frac{| | {Proj}_{1_{n}} X | |}{| | {Proj}_{1_{n}}^{⊥} X | |} \sqrt{n - 1} sgn (\overset{―}{X}) = \frac{R}{\sqrt{1 - R^{2}}} .$

2.1 Change of Basis

Let $Q = [\begin{matrix} | & | & | \\ q_{1} & q_{2} & \dots & q_{n} \\ | & | & | \end{matrix}] = n [\begin{matrix} | \\ q_{1} & Q_{r} \\ | \end{matrix}],$ where $q_{1} = \frac{1}{\sqrt{n}} 1_{n}$ , $q_{2}, \dots, q_{n}$ are complete orthonormal basis (like via Gram-Schmidt). So the new basis is $Z = Q^{T} X = (\begin{matrix} q_{1}^{T} X \\ Q_{r}^{T} X \end{matrix}) = (\begin{matrix} \sqrt{n} \overset{―}{X} \\ Q_{r}^{T} X \end{matrix}) .$

| | Q_{r}^{T} X | |^{2} = | | Q^{T} X | |^{2} - | | q_{1}^{T} X | |^{2} = | | X | |^{2} - n {\overset{―}{X}}^{2} = (n - 1) S^{2} .

\begin{aligned} Z_{1} \sim N (\sqrt{n} μ, σ^{2}), Z_{r} = Q_{r}^{T} X \sim N (0, σ^{2} I_{n - 1}) \\ \Rightarrow & S^{2} = \frac{1}{n - 1} | | Z_{r} | |^{2} \sim \frac{σ^{2}}{n - 1} χ_{n - 1}^{2} ⊥ ⊥ Z_{1} . \end{aligned}

(We already know from Basu's theorem)

3 Canonical Linear Model

Assume $Z = \begin{matrix} ^{d_{0}} \\ ^{d_{1} = d - d_{0}} \\ ^{d_{r} = n - d} \end{matrix} (\begin{matrix} Z_{0} \\ Z_{1} \\ Z_{r} \end{matrix}) \sim N_{n} ((\begin{matrix} μ_{0} \\ μ_{1} \\ 0 \end{matrix}), σ^{2} I_{n}) .$
Here $μ_{0} \in R^{d_{0}}, μ_{1} \in R^{d_{1}}, σ^{2} > 0$ . Test $H_{0} : μ_{1} = 0 vs H_{1} : μ_{1} \neq 0.$ Then $z$ is exponential family $p (z) = e^{{\frac{μ_{1}}{σ^{2}}}^{T} Z_{1} + {\frac{μ_{0}}{σ^{2}}}^{T} Z_{0} - \frac{1}{2 σ^{2}} | | Z | |^{2}} .$

$σ^{2}$ known, $d_{1} = 1$ , condition on $Z_{0}$ , reject for large/small/extreme $Z_{1}$ . Since $Z_{0} ⊥ ⊥ Z_{1}$ , test statistic is $Z_{1} \sim N (μ_{1}, σ^{2})$ , i.e. $\frac{Z_{1}}{σ} \overset{H_{0}}{\sim} N (0, 1)$ . (z-test)
$σ^{2}$ known, $d_{1} \geq 1$ , reject for large $| | Z_{1} | |$ . $\frac{| | Z_{1} | |^{2}}{σ^{2}} \sim χ_{d_{1}}^{2}$ . (chi^2 test)
$σ^{2}$ unknown, $d_{1} = 1$ : condition on $Z_{0}$ , $| | Z | |^{2} = | | Z_{1} | |^{2} + | | Z_{0} | |^{2} + | | Z_{r} | |^{2}$ . Reject for large/small/extreme $Z_{1}$ , i.e. reject for large $\frac{Z_{1}}{| | Z | |}$ , i.e. reject for large $\frac{Z_{1}}{\sqrt{| | Z_{r} | |^{2} / (n - d)}} \overset{H_{0}}{\sim} t_{d_{r}}$ . (t-test)
$σ^{2}$ unknown, $d_{1} \geq 1$ , reject for (conditionally) large $| | Z_{1} | |^{2}$ , i.e. reject for large $\frac{| | Z_{1} | |^{2} / d_{1}}{| | Z_{r} | |^{2} / (n - d)} \overset{H_{0}}{\sim} F_{d_{1}, n - d} .$ (F-test) Here $\frac{| | Z_{r} | |^{2}}{d_{r}} \sim \frac{σ^{2}}{d_{r}} χ_{n - d}^{2}$ functions as estimator of $σ^{2}$ , with $E {\hat{σ}}^{2} = σ^{2}, Var ({\hat{σ}}^{2}) = \frac{2 σ^{2}}{n - d}$ .

Compare:

Z	t	$χ^{2}$	F
$\frac{Z_{1}}{σ}$	$\frac{Z_{1}}{\hat{σ}}$	$\frac{\| Z_{1} \|^{2}}{σ^{2}}$	$\frac{\| Z_{1} \|^{2} / d_{1}}{{\hat{σ}}^{2}}$

3.1 Intervals for Canonical Model

How to test $H_{0} : μ_{1} = μ_{1}^{0} \in R^{d}$ ? The problem is $μ_{1}$ is not a natural parameter. We translate the problem to $(\begin{matrix} Z_{0} \\ Z_{1} - μ_{1}^{0} \\ Z_{r} \end{matrix}) \sim N_{d} ((\begin{matrix} μ_{0} \\ μ_{1} - μ_{1}^{0} \\ 0 \end{matrix}), σ^{2} I_{n}) .$
Can do some tests with $Z_{1} - μ_{1}^{0}$ replacing $Z_{1}$ . Invert:

$d_{1} = 1, σ^{2}$ known: $\frac{Z_{1} - μ_{1}}{σ} \sim N (0, 1)$ , so CI (confidence interval) is $Z_{1} \pm σ z_{α / 2}$ .
$d_{1} = 1, \hat{σ}$ unknown: $\frac{Z_{1} - μ_{1}}{\hat{σ}} \sim t_{n - d}$ , so CI is $Z_{1} \pm \hat{σ} t_{n - d} (\frac{α}{2})$ .
$d_{1} \geq 1, σ^{2}$ known: $\frac{| | Z_{1} - μ_{1} | |}{σ} \sim χ_{d_{1}}^{2}$ , so CI is $Z_{1} + σ \sqrt{c_{χ^{2}} (α)} B_{1} (0)$ .
$d_{1} \geq 1$ , $\hat{σ}$ unknown: $\frac{| | Z_{1} - μ_{1} | |}{\hat{σ}} \sim F_{d_{1}, n - d}$ , so CI is $Z_{1} + \hat{σ} \sqrt{c_{F} (α)} B_{1} (0)$ .

4 General Linear Model

Many problems can be put into canonical linear model after change of basis.
Basic setup: observer $Y \sim N_{n} (θ, σ^{2} I_{n}), σ^{2} > 0$ known or unknown. Test $H_{0} : θ \in Θ_{0} vs H_{1} : θ \in Θ ∖ Θ_{0},$ where $Θ_{0} \subset Θ$ are subspaces of $R^{n}$ , $\dim (Θ_{0}) = d_{0}, \dim (Θ) = d = d_{0} + d_{1}$ .
The idea is to rotate into canonical form.
For $Q =^{n} [\overset{d_{0}}{\overset{⏞}{Q_{0}}} \overset{d_{1}}{\overset{⏞}{Q_{1}}} \overset{n - d}{\overset{⏞}{Q_{r}}}],$ where $Q_{0}, Q_{1}, Q_{r}$ are respectively orthonormal basis for $Θ_{0}, Θ \cap Θ_{0}^{⊥}, R^{n} \cap Θ^{⊥}$ , and $Z = Q^{T} Y \sim N_{n} ((\begin{matrix} Q_{0}^{T} θ \\ Q_{1}^{T} θ \\ 0 \end{matrix}), σ^{2} I_{n}) .$ So $H_{0} : Q_{1}^{T} θ = 0$ . Like canonical linear model, we can do z, $χ^{2}$ , t, F test as appropriate.

Example (Linear Regression)

$x_{i} \in R^{d}$ fixed, $ε_{i} \overset{i . i . d}{\sim} N (0, σ^{2})$ . $y_{i} = x_{i}^{T} β + ε_{i}$ , $Y \sim N_{d} (X β, σ^{2} I_{n})$ , and denote $X = (\begin{matrix} - x_{1}^{T} - \\ ⋮ \\ - x_{n}^{T} - \end{matrix}) = (\begin{matrix} | & | \\ X_{1} & \dots & X_{d} \\ | & | \end{matrix}) \in R^{d \times n} .$ Assume $X$ has full column rank. $θ = X β \in Θ = Span (X_{1}, \dots, X_{d})$ which is the model space.
$H_{0} : β_{1} = \dots = β_{d_{1}} = 0$ ( $1 \leq d_{1} \leq d$ ), which is equivalent to $θ \in Θ_{0} = Span (X_{1}, \dots, X_{d_{0}})$ . Rotate into canonical basis $Q_{0}, Q_{1}, Q_{r}$ , and denote F-statistic $\frac{| | Q_{1}^{T} Y | |^{2} / d_{1}}{| | Q_{r}^{T} Y | |^{2} / (n - d)}$ , t-statistic $\frac{q_{1}^{T} Y}{\sqrt{| | Q_{r}^{T} Y | |^{2} / (n - d)}}$ (if $d_{1} = 1$ ).
It is nice to get more explicit expressions: $\begin{aligned} {\hat{β}}_{OLS} & = \arg min_{β} | | Y - X β | |^{2} = (X^{T} X)^{- 1} X^{T} Y, \\ | | Z_{r} | |^{2} & = | | Q_{r}^{T} Y | |^{2} = | | {Proj}_{Θ}^{⊥} (Y) | |^{2} = | | Y - {Proj}_{Θ} (Y) | |^{2} \\ = \sum_{i = 1}^{n} (y_{i} - x_{i}^{T} {\hat{β}}_{OLS})^{2}, \end{aligned}$ we call $r_{i} = y_{i} - x_{i}^{T} {\hat{β}}_{OLS}$ residual, and $| | Z_{r} | |^{2} = \sum_{i = 1}^{n} r_{i}^{2}$ residual sum of squares (RSS). $n - d$ is called residual degree of freedom.
Further $| | Z_{1} | |^{2} + | | Z_{r} | |^{2} = | | [Q_{1} Q_{r}]^{T} Y | |^{2} = | | {Proj}_{Θ_{0}}^{⊥} Y | |^{2} = | | {Proj}_{Θ_{0}}^{⊥} (Y) | |^{2}$ is called ${RSS}_{0}$ . The F-statistic is $\frac{| | Z_{1} | |^{2} / (d - d_{0})}{| | Z_{r} | |^{2} / (n - d)} = \frac{({RSS}_{0} - RSS) / (d - d_{0})}{RSS / (n - d)} .$
When $d_{1} = 1$ , let $X_{0} = (X_{2}, \dots, X_{d}) \in R^{d_{0} \times n}$ , and $q_{1} = \frac{X_{1 ⊥}}{| | X_{1 ⊥} | |}$ , where $\begin{aligned} X_{1 ⊥} & = X_{1} - {Proj}_{Θ_{0}} (X_{1}) \\ = X_{1} - X_{0} (X_{0}^{T} X_{0})^{- 1} X_{0}^{T} X_{1} \\ = X_{1} - X_{0} γ . \end{aligned}$ Reparametrize $θ = X β$ to $θ = X_{1 ⊥} β_{1} + X_{0} \underset{δ}{\underset{⏟}{(β_{- 1} + γ β_{1})}} = [\begin{matrix} X_{1 ⊥} & X_{0} \end{matrix}] (\begin{matrix} β_{1} \\ δ \end{matrix}) \begin{matrix} ^{1} \\ ^{d - 1} \end{matrix} .$
OLS solution in new parametrization is $\begin{aligned} (\begin{array}{c} {\hat{β}}_{1} \\ \hat{δ} \end{array}) & = ({[\begin{array}{c} X_{1 ⊥} & X_{0} \end{array}]}^{T} [\begin{array}{c} X_{1 ⊥} & X_{0} \end{array}])^{- 1} {[\begin{array}{c} X_{1 ⊥} & X_{0} \end{array}]}^{T} Y \\ = {[\begin{array}{c} X_{1 ⊥} & 0 \\ 0 & X_{0}^{T} X_{0} \end{array}]}^{- 1} (\begin{array}{c} X_{1 ⊥}^{T} Y \\ X_{0}^{T} Y \end{array}) = (\begin{array}{c} X_{1 ⊥}^{T} y / | | X_{1 ⊥} | |^{2} \\ (X_{0}^{T} X_{0})^{- 1} X_{0}^{T} Y \end{array}) . \end{aligned}$ Let $\hat{β_{1}} = \frac{X_{1 ⊥}^{T} Y}{| | X_{1 ⊥} | |^{2}}$ , so $se ({\hat{β}}_{1}) = \sqrt{Var ({\hat{β}}_{1})} = \frac{σ}{| | X_{1 ⊥ | |}}$ . T-statistic is $\frac{q_{1}^{T} Y}{\sqrt{RSS / (n - d)}} = \frac{{\hat{β}}_{1}}{\hat{σ} / | | X_{1 ⊥} | |} = \frac{{\hat{β}}_{1}}{\hat{se} ({\hat{β}}_{1})} .$

Example (Two-sample t-test & equal variance)

$Y_{1}, \dots, Y_{m} \overset{i . i . d}{\sim} N (μ, σ^{2})$ , $Y_{m + 1}, \dots, Y_{n + m} \overset{i . i . d}{\sim} N (ν, σ^{2})$ . Model: $θ = E Y = (\begin{matrix} μ 1_{m} \\ ν 1_{n} \end{matrix}) ⟺ θ \in Span ((\begin{matrix} 1_{m} \\ - 1_{n} \end{matrix}), 1_{n + m}) .$ Test $H_{0} : μ_{1} = μ_{2} ⟺ θ \in Span (1_{n + m})$ . $d_{0} = 1, d = 2, d_{r} = n + m - 2$ . Orthogonalize: $(\begin{matrix} 1_{m} \\ - 1_{n} \end{matrix}) \to (\begin{matrix} 1 / m \\ ⋮ \\ 1 / m \\ - 1 / n \\ ⋮ \\ - 1 / n \end{matrix}),$ and reject for large $\frac{\frac{1}{m} \sum_{i \leq m} Y_{i} - \frac{1}{n} \sum_{i > m} Y_{i}}{\sqrt{\frac{1}{m} + \frac{1}{n}} \cdot \sqrt{\frac{RSS}{n + m - 2}}} = \frac{{\overset{―}{Y}}_{1} - {\overset{―}{Y}}_{2}}{\hat{σ} \sqrt{\frac{1}{m} + \frac{1}{n}}} .$

Example (One-way ANOVA)

ANOVA (Analysis of Variance) is used to test if multiple ( $\geq 3$ ) groups have significant difference in group means.

$Y_{k, i} \overset{i . n . d}{\sim} μ_{k} + ε_{k, i}$ , $ε_{k, i} \overset{i . i . d}{\sim} N (0, σ^{2})$ , $k = 1, \dots, m, i = 1, \dots, n$ . $H_{0} : μ_{1} = \dots = μ_{m} = μ$ .
${\overset{―}{Y}}_{k} = \frac{1}{n} \sum_{i = 1}^{n} Y_{k, i}$ , $S_{k}^{2} = \frac{1}{n - 1} \sum_{i = 1}^{n} (Y_{k, i} - {\overset{―}{Y}}_{k})^{2}$ ; $\overset{―}{Y} = \frac{1}{m n} \sum_{k = 1}^{m} \sum_{i = 1}^{n} Y_{k, i}$ , $S_{0}^{2} = \frac{1}{m n - 1} \sum_{k = 1}^{m} \sum_{i = 1}^{n} (Y_{k, i} - \overset{―}{Y})^{2}$ .
$d_{0} = 1, d = m, d_{r} = m (n - 1)$ . So $RSS = \sum_{k, i} (Y_{k, i} - {\overset{―}{Y}}_{k})^{2} = | | Y | |^{2} - n \sum_{k = 1}^{m} {\overset{―}{Y}}_{k}^{2},$ ${RSS}_{0} = \sum_{k, i} (Y_{k, i} - \overset{―}{Y})^{2} = | | Y | |^{2} - m n {\overset{―}{Y}}^{2},$ ${RSS}_{0} - RSS = n (\sum_{k = 1}^{m} {\overset{―}{Y}}_{k}^{2} - m {\overset{―}{Y}}^{2}) = n \sum_{k = 1}^{m} ({\overset{―}{Y}}_{k} - \overset{―}{Y})^{2} .$ F-statistic is $\frac{\frac{n}{m - 1} \sum_{k = 1}^{m} ({\overset{―}{Y}}_{k} - \overset{―}{Y})^{2}}{\frac{1}{m (n - 1)} \sum_{k = 1}^{m} \sum_{i = 1}^{n} (Y_{k, i} - {\overset{―}{Y}}_{k})^{2}} .$